…s and auto-enable split mode for CEF logs

… formats for more reliable extraction

…ue logs instead of truncating prefixes

…source pattern learning

…ders from all logs

…tterns and full preceding words instead of truncating prefixes

…d generalize them to \d+ to prevent brittle anchors

…on for decoders

…ecoder prefixes

…ation

- Removed Decoder Generator and Rule Generator sections from HTML
- Moved input fields (appName, logsInput, extractFields, etc.) into AI view
- Removed 'Generate Decoder' and 'Generate Rule' sidebar nav items
- Made 'AI Generate' the default active view
- Cleaned up app.js: removed unused functions (showAnalysis, showXml,
  syncFeedback, readRulePayload, rule conditions UI, old button handlers)
- Updated history loading and test function to work without decoder view

- Added POST /api/install endpoint to write decoder/rule XML to Wazuh's
  custom decoders/rules directories (SSH or local)
- Added POST /api/uninstall endpoint to remove installed files
- Added POST /api/logtest/raw endpoint for running wazuh-logtest with
  arbitrary log samples and returning raw output + parsed fields
- Redesigned Test view with three cards: Installed Decoder (install/
  uninstall), Test Logs (editable sample input), and wazuh-logtest
  Output (raw stdout + parsed fields table)
- Added state management storing installed file paths in localStorage
- AI-generated XML is now persisted in JS so it can be installed from
  the Test view without re-running AI generation

…ailure

- Add generation_mode (auto/decoder_only/rule_only/both) to AI request
- Add validate_with_logtest flag and /api/ai/generate-validated endpoint
- Add _collect_ai_response, _extract_xml_from_ai_response helpers
- Add _validate_ai_decoder_with_logtest for auto-install+test validation
- Refactor _build_ai_prompt: shorter config block, concise ML/logtest context
- Add system prompt for Ollama (system+user roles), fix URL path
- Lower default temperature to 0.05 for more deterministic output
- Default model changed to wazuh-decoder
- UI: generation mode dropdown, validate checkbox, Generate & Validate button
- UI: show validation badge & details in AI output section
- UI: hide rule section when generation_mode=decoder_only

…ndpoint and automate rule group/static field sanitization

…coring, and sigmoid calibration

- Add log-type detection (_detect_log_type) with type-based boosting to bias results toward relevant decoder families (JSON, Windows, syslog, etc.)
- Add regex token overlap scoring (_regex_overlap_score) to boost patterns whose OS_Regex tokens match query log literals
- Add sigmoid confidence calibration for well-calibrated probabilities in [0,1]
- Tune ensemble weights: TF-IDF 0.3, SBERT 0.7 (semantic model is stronger for unseen formats)
- Raise minimum confidence gate to 0.15 to avoid low-confidence noise
- Add fine-tuned SBERT checkpoint loading with graceful fallback
- Enhance tokenizer to preserve more OS_Regex character classes

… Modelfile

- Lower temperature (0.05→0.02) and top_p (0.85→0.80) for more deterministic output
- Increase repeat_penalty (1.15→1.20) and lower top_k (20→15) to reduce repetition
- Add self-validation checklist to catch common errors before output
- Add JSON log decoder and DHCP/MAC address examples
- Fix sshd example to use same decoder name for multiple children
- Add instruction: 'No text before or after' the XML block

…lization

- Default OLLAMA_BASE_URL to http://localhost:11434/v1 so it works without env vars
- Normalize /v1 suffix to prevent double-/v1 404 errors in URL construction
- Add 60s timeout to streaming client with retry on ReadTimeout (up to 3 attempts)
- Add decoder rule: multiple child decoders must use exact same decoder name
- Fix IP regex guidance: do not escape dots in \d+.\d+.\d+.\d+
- Update top_k to 15 and repeat_penalty to 1.20 to match Modelfile tuning
- Improve error messages for network/server issues

…to dataset builder

- Add load_rejection_records(): convert rejection notes with regex corrections into positive training pairs
- Add augment_with_dropout(): create robustness variants by randomly masking log tokens (15% prob)
- Rejection corrections teach SBERT to distinguish correct from broken regex patterns
- Dropout augmentation teaches model that partial log lines still map to same decoder
- Add structured logging of record counts throughout pipeline

…nting to SBERT training

- 5 epochs with best-checkpoint saving (by validation AUC)
- Larger batch size (64 configurable) for better in-batch negatives with MultipleNegativesRankingLoss
- Hard-negative augmentation: pair logs with categorically distinct decoders (30% ratio)
- Token dropout data augmentation for robustness on partial input
- Early stopping with patience=2 epochs
- Add binary evaluator with both positive and negative pairs for AUC measurement
- Configurable training device (default CPU to avoid MPS OOM with Ollama)
- Copy best checkpoint to 'final' directory for easy model loading

The sidebar defaulted to AI Generate as active, but the corresponding
#view-ai div was missing the 'active' class, so CSS display:none kept
the entire AI generation page blank on initial load.

…egex instruction

The AI model consistently escapes dots (\.) in regex patterns because it is
trained on PCRE where this is correct. Wazuh OS_Regex treats '.' as a literal
character, so \. is wrong syntax.

Fix:
- Add _sanitize_decoder_xml_osregex() that strips \. → . in generated XML
- Apply it in _extract_xml_from_ai_response and the final return
- Strengthen the Modelfile and prompt instruction with WRONG/RIGHT examples
  to make the rule impossible to miss

- Fix sanitization regex: r'\.' was matching any char after backslash
  (breaking \d, \w, etc.). Use r'\\.' to match only backslash + literal dot.
- Add Example 7 to Modelfile showing correct TrafficLog IP extraction
  with unescaped dots in OS_Regex
- Strengthen prompt WRONG/RIGHT examples for IP regex

The streaming /api/ai/generate endpoint returns raw AI text without
server-side processing, so escaped dots (\.) pass through to the browser.
Add sanitizeOsRegex() in app.js that strips \. → . client-side after
XML extraction, covering both the streaming and validated endpoints.

Add a hand-curated example with unescaped dots for IP regex
(\d+.\d+.\d+.\d+) so the fine-tuned model natively learns
correct OS_Regex IP syntax instead of relying on post-processing.

…gex instructions

- Add _stream_ai_sanitized to post-process AI output and fix \d+\.\d+ → \d+.\d+
  (common AI mistake: escaping dots before \d for IPs in Wazuh OS_Regex)
- Enhance _sanitize_decoder_xml_osregex to target IP patterns specifically,
  only removing \. between \d quantifiers, not valid \.+ any-char quantifiers
- Update Modelfile with clearer IP regex instructions and new example conversations
- Update Modelfile.finetune with more training examples (iptables, squid, UFW, TrafficLog,
  CEF Palo Alto, nginx, SSH, netfilter, KV log)
- Fix kv-log-fields decoder order to match extract_fields

…ex sanitizer

- Remove programmatic XML from ai_generate and ai_generate_validated
  — AI now generates from scratch using only analysis context
- Add _fix_osregex_bare_dot_quantifier: converts common AI mistakes
  (.+) → (\S+), .+ → \.+, .* → \.+ inside regex/prematch tags
- Update _OLLAMA_SYSTEM_PROMPT with explicit anti-pattern examples
  showing CORRECT vs WRONG OS_Regex patterns
- Strengthen _build_ai_prompt decoder_rules with OS_Regex constraints
  and anti-echo instructions
- Update Modelfile and Modelfile.finetune with anti-pattern section

…d bare-dot sanitizer

- Remove raw streaming output (#aiOut) from UI — only show final extracted XML
- Add Reference Field-to-Pattern Mapping in _build_ai_prompt: programmatic
  regex patterns as text guidance (not XML blocks AI can echo)
- Add _infer_osregex_type helper to suggest correct OS_Regex pattern per field
- Add _build_fallback_decoder: silently builds programmatic decoder when AI
  produces no valid XML (uses user inputs like field_hints)
- Add _fix_osregex_bare_dot_quantifier: sanitizes (.+) → (\S+), .+ → \.+,
  .* → \.+ inside regex/prematch tags
- Update ai_generate_validated to fall back when all retries fail
- Remove ai-stream-block CSS (unused)

…d-aid sanitization

AI now handles structure (decoder names, hierarchy, order tags) but regex
patterns come from the proven programmatic engine. _inject_programmatic_regex
matches each <regex> to the next <order> and replaces the content with the
correct regex from analysis regex_order_pairs.
- Remove unreliable bare-dot/IP sanitizers for regex content
- _extract_xml_from_ai_response accepts regex_order_pairs param
- ai_generate and ai_generate_validated pass analysis data through

- Remove _inject_programmatic_regex, _build_fallback_decoder
- Remove _FIELD_PATTERN_MAP, _infer_osregex_type
- Remove Reference Field-to-Pattern Mapping from _build_ai_prompt
- _sanitize_decoder_xml_osregex reverts to band-aid fixes only
- _extract_xml_from_ai_response no longer takes regex_order_pairs
- ai_generate and ai_generate_validated have zero programmatic fallback
AI generates everything (structure + regex) independently.

…ript

AI now generates everything (structure + regex) independently.
Only band-aid sanitization remains: (.+) → (\S+), .+ → \.+,
\d+\.\d+ → \d+.\d+.

Add scripts/train_osregex.py — extracts 26 training pairs from
Modelfile.finetune into JSONL format and provides training commands
for Ollama 0.5+, Unsloth, llama.cpp, and Axolotl.

…regex correction

- scripts/generate_finetuning_data.py: downloads all 104 decoder + 133
  rule XMLs from wazuh-ruleset, generates 806 training pairs
  (725 train / 81 val) in JSONL format
- app/main.py: _inject_correct_regex silently replaces AI <regex>
  content with analysis-derived patterns; _INTERNAL_FIELD_REGEX
  maps field names to correct OS_Regex
- scripts/train_osregex.py: points to new 806-example dataset
- .gitignore: add .cache_decoders/

…refix regex generation

- sanitizeOsRegex disabled: backend _inject_correct_regex already fixes patterns
- build_split_regexes_from_fields: better first-field prefix handling (no \.+ prefix for start-of-log fields); use (\.+) for multi-word/multi-token field values

…ead of preceding words

…on; expand CEF field aliases

- app.js: checkExistingDecoder() calls /api/analyze first and shows
  confirm() dialog if a builtin decoder already matches
- main.py: add 'source', 'destination', 'port' aliases for CEF field mapping

…gex token patterns

- AI prompt: provide correct prematch when no program_name is pre-decoded
- AI prompt: add both <program_name> and <prematch> strategy examples
- decoder_ml_enhanced.py: fix over-escaped regex tokens in enhanced tokenizer
- wazuh_logtest.py: use env vars with fallback defaults instead of hardcoded values
- .gitignore: exclude certs/, *.pem, *.key, *.crt
- README.md: make SSH config docs generic